arXiv:1507.06647v2 [cond-mat.mes-hall] 25 Dec 2015 


Spontaneous symmetry breaking and quantum Hall valley ordering on the surface of topological 

hexaborides 

Xiao Li,* Bitan Roy, and S. Das Sarma 
Condensed Matter Theory Center and Joint Quantum Institute, 

University of Maryland, College Park, Maryland 20742-4111, USA 
(Dated: December 29, 2015) 

A number of strongly correlated heavy fermion compounds, such as samarium (Sm), ytterbium (Yb), plu¬ 
tonium (Pu) hexaboride, are predicted to become topological insulators at low temperatures. These systems 
support massless Dirac fermions near certain (three) points of the surface Brillouin zone, hereafter referred to 
as the valleys. In strong perpendicular magnetic fields, the conical Dirac dispersions of these surface states 
quench onto three sets of Landau levels and we predict various possible hierarchies of incompressible quantum 
Hall states on the surface of hexaborides. In addition, we address the effects of strong electron-electron inter¬ 
action within the surface zeroth Landau levels. Specifically, we show that depending on the relative strength 
of the long-range (Coulomb-type) and the finite-range (Hubbard-type) interactions the ground state can display 
either a valley-polarized or a valley-coherent distribution of electronic density. We also show that the transi¬ 
tion between two valley-polarized states is always discontinuous, while that between a valley-polarized and a 
valley-coherent phase is continuous. The Zeeman splitting and/or an applied uniaxial strain on the surface can 
drive the system through various quantum phase transitions and place it in different broken-symmetry phases. 

Application of uniaxial strain is also shown to considerably modify the precise sequence of quantum Hall states. 

We also highlight the role of topology in determining the broken symmetry phases, disorder on the surface of 
topological hexaborides in strong magnetic fields. 


I. INTRODUCTION 

Two-dimensional semiconductor heterostructures, such as 
those prepared in GaAs quantum wells and Si metal- 
oxide-semiconductor field-effect transistors (MOSFETs), of¬ 
fer unique opportunities to study many novel properties of 
low-dimensional electronic systems. Possibly the most cele¬ 
brated example is the quantum Hall effect [1,2], where precise 
quantization of the Hall conductance is observed at low tem¬ 
peratures in the presence of a strong perpendicular magnetic 
field. The quantum Hall states represent the first example of 
topological states of matter [time-reversal symmetry break¬ 
ing due to the presence of an external magnetic field, but pro¬ 
tected by U(l) charge conservation symmetry] with a bulk gap 
and one-dimensional chiral edge modes at the boundary. The 
ground state topology of integer quantum Hall states is cap¬ 
tured by the Thouless-Kohmoto-Nightingale-de Nijs (TKNN) 
invariant or the first Chem number [3]. 

In the past decade, our notion of topological states of mat¬ 
ter has been vastly expanded. It began with the generalization 
of topological vacua protected by time-reversal invariance, 
which led to the discovery of two- and three-dimensional 
topological insulators and superconductors [4-7]. In fact, 
almost immediately after the theoretical proposal, Bi 2 Se 3 
was established as the first prototype candidate for three- 
dimensional strong Z 2 topological insulator [8, 9]. These de¬ 
velopments stimulated a surge of theoretical works, aimed 
at extracting topological information from various quantum 
phases of matter [10-13]. 

From the perspective of quantum field theory, a topologi¬ 
cally nontrivial ground state can be described as a ^-vacuum. 
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and different values of 6 correspond to topologically distinct 
phases. One well studied example of such a topological quan¬ 
tum field theory is the Chern-Simons theory in two dimen¬ 
sions, where the coefficient of the ^-term dictates the Hall con¬ 
ductivity in the system [14, 15]. In three dimensions, the cor¬ 
responding topological field theory is tied with the magneto¬ 
electric effect, captured by an axion term ^axE • B [16]. The 
axion angle for a time-reversal and inversion symmetric strong 
Z 2 topological insulator is ^ax = (2^ - l)7r, where n is an inte¬ 
ger ^ . A nontrivial axion angle (modulo In) leaves its signa¬ 
ture in the bulk, where the insulation through the band inver¬ 
sion occurs at 2^ - 1 high-symmetry time-reversal invariant 
momentum (TRIM) points of the Brillouin zone. Through the 
bulk-boundary correspondence, the surface hosts 2^-1 copies 
of two-component massless Dirac fermions. Angle-resolved- 
photo-emission-spectroscopy (ARPES) [8, 9, 18-20], scan¬ 
ning tunneling microscopy (STM) [21] and quantum oscilla¬ 
tion measurements [22] have confirmed the existence of such 
topologically protected Dirac cones on the surface of various 
topological insulators. In Bi 2 Se 3 , for example, the bulk band 
inversion occurs only at the F point. As a result, the surface 
state is comprised of only a single copy of massless helical 
Dirac cone, carrying a nontrivial Berry phase n. 

Recently a family of hexaboride compounds, such as 
SmB6 [23-35], YbB6 [36-39] andPuB6 [40], together named 
topological hexaborides (THBs), ^ have been predicted to 
be topological insulators at low temperatures. As shown in 
Fig. 1, these systems have cubic unit cells, with six boron 
atoms forming a cage at its center, while one A atom (A=Sm, 


^ We note that when the ground state has a D-fold degeneracy, the axion 
angle is allowed to take fractional values 0ax ^ ^ax/D [17]. However, we 
do not delve into such a situation. 

^ SmB6, PuB 6 are also referred to as “topological Kondo insulators” 
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FIG. 1. (a) Crystal structure of topological hexaborides (ABe). Here, 
B represents boron atoms, six of wbicb form a cage at tbe center of 
tbe cubic lattice, and A=Sm, Yb, Pu, for example, (b) Bulk Brillouin 
zone for AB6 and its projection on (001) surface. Tbe bulk band 
inversion takes place at three high-symmetry points X - {n, 0,0), 
Y - (0, 71, 0), and Z = (0,0, ti) (marked by blue dots). As a result, the 
(001) surface hosts three Dirac cones (marked by red dots). 


Yb, Pu) is shared by eight corners. In the cubic environment 
the band inversion takes place at three X points in the bulk 
Brillouin zone, and is labeled as Y - p3(4)x in the space 
group classification of topological insulators [41]. Conse¬ 
quently each surface accommodates three copies of gapless 
Dirac cones. On the high symmetry (001) surface, for exam¬ 
ple, the Dirac points are located at f = (0,0), A = (tt, 0), 
and Y = (0,7r), as shown in Figs. 1(b) and 2(a). Therefore, 
THBs can be considered as topological insulators with an ax- 
ion angle ^ax = 3;:, and they belong to the same class All 
in ten-fold way of classification as Bi 2 Se 3 [12]. Furthermore, 
the chemical potential in THBs resides within the bulk insu¬ 
lating gap, offering rare examples of true topological insula¬ 
tors, unlike the situation in typical semiconductor topological 
insulators, such as Bi 2 Se 3 which inevitably tend to have con¬ 
siderable bulk conduction due to inherent doping induced by 
defects. The insulating bulk nature of THBs, along with grow¬ 
ing experimental evidence for a two-dimensional (2D) surface 
conducting layer, has led to much activities in these systems. 

In contrast to conventional (weakly correlated) topological 
insulators such as Bi 2 Se 3 , the bulk gap in THBs stems from 
strong interaction-driven hybridization between the Kramers 
degenerate bands with opposite parities. For example, the 
hybridization predominantly takes place between d and / or¬ 
bitals in SmB6 and PuB6, and between p orbitals in YbB6. 
As a result, strong electronic interactions, aside from giving 
rise to topologically nontrivial bulk insulators, may as well 
lead to interesting many-body phenomena on the surface [42- 
47]. Therefore when placed in strong magnetic fields, the 
surface of THBs turns into a fertile ground for exploring the 
interplay of topology, electronic interactions, and the quan¬ 
tum Hall phenomena, both theoretically and experimentally. 
Recent success in experimentally observing surface quantum 
Hall states in conventional topological insulators [48, 49] and 
the presence of strong residual interaction on the surface of 
THBs, motivates this work. We here study various possible 
broken-symmetry quantum Hall phases in two-dimensional 
metallic surface of THBs. 


Recent experiments suggest that certain hexaboride sys¬ 
tems, in particular SmB6 and YbB6, can become topological 
insulators (i.e., with a bulk gap and an odd number of sur¬ 
face Dirac cones) at low temperatures. For example, recent 
ARPES experiments have confirmed the existence of three 
Dirac cones around f, X, Y points of the surface Brillouin 
zone, and the spin-resolved ARPES is strongly suggestive of 
their spin-momentum locking [28-33] (the hallmark signature 
of surface states in strong Z 2 topological insulators). ARPES 
has further exposed that while the Dirac cone at f is circular, 
those near X and Y are elliptic. Signature of surface metallic 
states has also been observed in STM measurements [27]. In 
addition, De Haas-van Alphen effect [50] has found traces for 
two distinct sets of Dirac-type Landau levels up to 45 Tesla 
(one from the f pocket, and other one from the X and the 
Y pockets together). An infinite-field {B 00 ) extrapola¬ 
tion from these data reveals a -71 Berry phase for each such 
Dirac cone. These observations are in agreement with recent 
theoretical analyses that predict a generic off-set between the 
energies of the Dirac points at f {Ef) and X, Y (E’l/?) points. 
Such an outcome can also be anticipated based on a symme¬ 
try analysis: while the underlying cubic symmetry mandates 
Ex = Ey on the (001) surface, Ey remains unconstrained and 
typically \Ef - £’x/?l = Aq ~ 2 - 10 meV [51, 52]. Magne¬ 
toresistance measurements in a w^Jg^-shaped SmB6 sample 
showed that transport at low temperatures (T < 10 K) is inde¬ 
pendent of the sample thickness and dominated by the surface 
states [53]. These experimental observations, when combined 
with recent tight-binding [51] and first-principles [52] calcu¬ 
lations, enable us to establish a minimal model [see Eq. (1)] to 
study the effects of strong magnetic field and electron-electron 
interaction on the surface of THBs. 

In this work we study the interplay of external magnetic 
field and strong Coulomb repulsion on the surface of THBs. 
Eollowing, we promote the central results of our theoretical 
study and the organization principle for the rest of the paper. 

(i) In Sec. II, we account for the effects of strong mag¬ 
netic field on the surface of THBs and present the spectrum 
of surface Landau levels. Using realistic values of effective 
band parameters, we show that the crossing between the ze¬ 
roth Landau levels and those at finite energies is avoided for 
magnetic fields beyond 5=, ~ 1 T, see Eig. 2(b). Perturbations 
from Zeeman coupling and/or uniaxial strain do not change 
this result quantitatively. Eurthermore, we show that depend¬ 
ing on the relative position and the anisotropy of these surface 
Dirac cones, a number of distinct hierarchies of quantum Hall 
states can be observed in these systems. 

(ii) Within the framework of a zeroth Landau level (LL) ap¬ 
proximation (i.e., neglecting LL mixing which are negligible 
at high magnetic fields), we develop the mean-field Hartree- 
Eock theory for interacting surface states in Sec. III. Both 
long-range (Coulomb-type) and short-range (Hubbard-type) 
interactions are accounted for in the theory, which should 
be exact in the limit of a very large inter-LL gap, since the 
Hartree-Eock theory is a systematic expansion in the ratio of 
interaction to the LL separation energy. This section is slightly 
technical and readers interested in the results may choose to 
skip the discussion. 
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(iii) The phase diagrams of interacting surface states in 
strong magnetic fields are discussed in Sec. IV. We show that 
for long-range Coulomb interactions the electronic density 
can only display valley polarizations. Due to the anisotropic 
cyclotron orbits near X and Y valleys, the long-range interac¬ 
tion prefers a f polarized state [see Fig. 6(a)], which does not 
break any microscopic symmetry. On the other hand, the X 
or F-valley polarized states break the discrete rotational sym¬ 
metry of the underlying lattice, and correspond to the quantum 
Hall Ising nematic order. When finite-range or backscattering 
interactions are included, a valley-coherent phase can also be 
realized [see Fig. 6(b)]. Finally, we show that various single¬ 
particle perturbations, such as the Zeeman coupling and the 
uniaxial strain serve as useful tuning parameters to drive the 
system through various quantum phase transitions and place 
the system in different broken-symmetry phases (see Figs. 7 
and 8). 

(iv) We summarize our findings and present concluding re¬ 
marks in Sec. V. In addition, we propose experimental signa¬ 
tures, such as the measurement of longitudinal conductivity 
{Rjj) and the scaling of XR = \Rxx - Ryy\ with magnetic field 
at finite temperatures that can pin the exact nature of the bro¬ 
ken symmetry states within the zeroth Landau level. The role 
of disorder in the ordered phases, and the importance of un¬ 
derlying topology in determining the ground states are also 
highlighted in this section. 


II. SURFACE STATES: EXTERNAL MAGNETIC AND 
STRAIN FIELDS 



FIG. 2. (a) Illustration of three electron pockets, and (b) the Landau 
level spectrum (for \N\ < 10) on the (001) surface of THBs. The 
black (blue) curves represent the Landau levels from the F (X or Y) 
valley. The energy offset between these two sets of Landau levels is 
Ao = 5 meV, the anisotropy factor is ^ = 0.2, and the Fermi velocities 
are chosen as Vf = 6.0 x 10^ m/s and Vx/f = 3.0 x 10^ m/s. 


a four-fold rotational symmetry. However, the surface state 
at the F valley in general has a different energy. The energy 
difference between the F and X/Y valleys can be tuned inde¬ 
pendently, for example, through band bending effects [51]. To 
capture this energy difference, we introduce a constant shift 
Ao to Hf. The last two terms SHf and arise from the 
Zeeman coupling and an external uniaxial strain, respectively, 
and will be discussed later. 


A. Surface Landau levels 


To set a context for our study, we start by introducing an ef¬ 
fective single-particle description on the surface of THBs. We 
focus on the high-symmetry (001) surface, which hosts three 
copies of massless Dirac fermionic excitations at F, X, and Y 
points of the 2D surface Brillouin zone, hereafter referred to 
as valleys, as shown in Fig. 2(a). The effective Hamiltonian 
for these surface states is given by [42, 51] 

Ha = hvx [(1 + 6A)k^a-x - (1 - Sx)kya-y'\ 

+ 6Hl + dHf, (1) 


In the presence of a uniform perpendicular magnetic field B, 
the surface states support three sets of Landau levels. For now 
we neglect the effects of Zeeman coupling and external strain. 
The Landau level spectrum can be found by taking the Peierls 
substitution hk ^ iz = hk + eA. In the Landau gauge the 
vector potential is given by A = (0, Bx), from which we define 
three pairs of creation and annihilation operators [54, 55] 


i , 

ax = —^(axTTx - I —), a, 
yflh ax 


,t _ 


-^iaxTTx + i—), (2) 

X2h ax 


where A = 1,2,3 corresponds to the F, X, and Y valley, re¬ 
spectively. The Fermi velocity of the surface Dirac fermions 
is taken to be vp = 6.0 x 10^ m/s and Vx/f = 3.0 x 10^ m/s 
for our numerical work, in accordance with recent quantum 
oscillation experiments [50]. However, note that quantum os¬ 
cillation experiments cannot pin the Fermi velocity associated 
with the Dirac cone at a given valley. Thus, we combine 
the information from quantum oscillation experiments with 
ARPES data [28-33], which suggests that the surface state 
near F point has a higher Fermi velocity. 

The parameter characterizes the anisotropic Fermi ve¬ 
locity for the surface states near each valley. For a pristine 
surface, we set dp = 0 (representing a circular Fermi surface 
near the f valley) and Sx = -Sf = S (accounting for elliptic 
Fermi surfaces near X and Y valleys). In addition, the surface 
states at X and Y valleys are energetically degenerate due to 


where ^ = sJhIeB is the magnetic length. The factor ac¬ 
counts for the anisotropy of the Fermi surface at each valley, 
with ax = V(1 + ^)/(l “ ^) = and ap = 1. In terms of 
these creation and annihilation operators, the surface Hamil¬ 
tonian (1) takes the form 




I 


0 a\ 
a^ 0 




( 3 ) 


for T = 1,2,3. The Landau level spectrum near each valley 
can then be readily obtained, yielding 

£w(f) = Ao + sgn {N) y^leExfivlm, 

En(XIY) = sgn (N) ^2eBxhvl\N\(l - 6^), (4) 
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where N = 0,1,2, • • • , is the Landau level index. The Lan¬ 
dau level spectrum for S = 0.2 and Aq = 5 meV is shown in 
Fig. 2(b). With these parameters the crossing between the ze¬ 
roth Landau levels and the remaining ones is avoided for fields 
beyond a critical value defined as 




2ehvl(l - S^) 


0.65 T. 


( 5 ) 


The absence of Landau level crossings at large fields will be¬ 
come crucial when we account for electron-electron interac¬ 
tions within the zeroth Landau level. 

The wave function for the Landau levels at each valley 
is given by 


e^( ^|TT6^o<p\^\(x-X) '' 

721 ; [d - Sn,o) sgn (N) - X)^ 


where Ly is the linear dimension of (001) surface in the y- 
direction, X is the guiding center coordinate, and 0^\v) is the 
harmonic oscillator wave-function, 


<Pn\x) = 




{2^N\ 




( 6 ) 


where is the Hermite polynomial of order N. Note that the 
wave function for the zeroth Landau level near each valley is 
spin-polarized. Therefore, on two opposite surfaces of a topo¬ 
logical insulator the zeroth Landau levels are projected onto 
identical spin components. In contrast, the components of 
the spinor basis in graphene are constituted by the pseudospin 
(sublattice) degrees of freedom, instead of the real spin. As 
a result, in graphene the zeroth Landau levels near two oppo¬ 
site valleys reside on complementary sublattices for each spin 
projection. 

Although we here assumed three valleys to be discon¬ 
nected, the bands always get connected to each other at high 
energy. However, such connectivity of the bands from three 
valleys does not spoil the existence of zeroth Landau levels, 
as the chiral nature of the dispersive surface states mandates 
the existence of three fold zeroth Landau level. The robust¬ 
ness of zeroth Landau level in the presence of a magnetic field 
of arbitrary profile (as long as a finite flux gets trapped by the 
system) stems from iht Aharonov-Casher index theorem [56], 
which remains operative even on a lattice [57]. 

So far we have assumed that surface Dirac points reside 
within the bulk insulating gap, which may not be the generic 
situation in reality. Within the framework of an effective 
description for TIs, the bulk gap can be considered as a 
continuous chiral symmetry breaking (but parity and time- 
reversal symmetry preserving) mass for three dimensional 
Dirac fermions. With the application of external magnetic 
fields, the amplitude for the chiral symmetry breaking mass 
increases monotonically [58], while the surface zeroth Lan¬ 
dau level continue to rest at the Dirac point. Therefore, if 
the surface Dirac points are buried within few meV of bulk 
valence band (in the absence of magnetic field), the surface 
zeroth LLs can be pulled out from the sea of three dimen¬ 
sional dispersive LLs, and our following discussion remains 



B (T) B (T) B (T) 


FIG. 3 . Lowest few Landau levels (\N\ < 3 ) for various possible 
effective band parameters for the surface states, (a) Aq = 0, Vf = 
Vx = 3 x 10 ^ m/s, and^ = 0 ; (b) Aq = 5 meV, = Vf /2 = 3 x 10 ^ m/s, 
and 6 = 0 . 2 ; (c) Aq = 5 meV, Vf - Vx/2 - 3 x 10 ^ m/s, and 6 - 0 . 2 . 
The solid and dashed lines represent Landau levels near the F and 
XjY valleys, respectively. As a result, the blue lines are all doubly 
degenerate, while the black ones are non-degenerate. 


justified in the strong magnetic fields limit. It should, how¬ 
ever, be noted that when surface Dirac points gets immersed 
deep inside the bulk valence band [say ~ lOOmeV, although 
such situation is quite pathological in the context of all known 
three dimensional topological insulators (TIs)], even strong 
magnetic field (~ 30 T) may not be sufficient to place the sur¬ 
face zeroth LL within the bulk gap. But, we do not delve into 
such unrealistic situation in this work. 


B. Quantum Hall states in topological hexaborides 

Having described the Landau level spectrum for the surface 
states, we can now discuss the possible quantum Hall states 
in this system. As we show in the following, depending on 
the relative position of the Dirac points and their Fermi ve¬ 
locities, different hierarchies of quantum Hall states can be 
realized on the surface of THBs. Thus, the sequence of Hall 
plateaus can provide valuable information about surface states 
of THBs. Remember that both top and bottom surfaces un¬ 
dergo Landau level quantization in the presence of a perpen¬ 
dicular magnetic field. Hence, all Hall plateaus we propose 
in the following should be augmented by an additional factor 
of 2. In addition, while the top and bottom surfaces host Lan¬ 
dau levels and thus chiral edge states, the four side surfaces 
accommodate nonchiral helical states (assuming the sample 
to be cubic). These states can destroy the perfect quantiza¬ 
tion of Hall conductivity [59-61]. However, recently clear 
signatures of Hall plateaus have been observed on the surface 
of Bi 2 Se 3 [49]. Therefore, we expect that our proposed Hall 
states can also be observed on the surface of SmB^ and YbBe, 
especially given that Dirac-type Landau levels have been ob¬ 
served for the surface states of SmB 6 [50]. 

Let us begin the discussion with a very simple situation: 
(i) Ao = 0, Vf = Vx and d = 0, for which the Landau 
level spectrum is shown in Fig. 3(a). Surface states then host 
three-fold degenerate Landau levels and the Hall plateaus ap- 
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pear at fillings |y| = 3 Q + where N = 0,1,.... (ii) 
Next we consider a realistic scenario with Aq = 5 meV, 
/2 = 3 X 10^ m/s, and S = 0.2, for which the Landau 
level spectrum is shown in Fig. 3(b). In this case the degen¬ 
eracy of Landau levels between the F valley and the X/Y val¬ 
ley is lifted, and the lowest few Hall plateau sequence will be 
y = ..., _(hi) Finally, we focus on the sit¬ 

uation when the F valley electrons have smaller Fermi veloci¬ 
ties, i.e., Ao = 5 meV, Vf = v^/2 = 3x10^ m/s, and S = 0.2. 
As shown in Fig. 3(c), the lowest few Hall plateau will show 

up for filling factors atv = _ 

Thus, from the sequence of Hall plateaus, one can pin the rel¬ 
ative location of the Dirac cones and their Fermi velocities. 

Throughout this paper, we will assume that the sample of 
TKIs is thick, and the system is far from the transition point 
between the topological and the trivial insulating phases, so 
that the overlap among the states residing on the top and the 
bottom surfaces is extremely weak. As a result, the total con¬ 
tribution to the Hall conductivity from the top and the bottom 
surfaces simply adds up. Thus, our analysis belongs to the 
opposite limit from the one in quantum Hall bilayer system, 
in which the interlayer tunneling is strong. By contrast, in the 
thin film limit the interlayer tunneling is strong and the anal¬ 
ysis will closely resemble to the one for bilayer Hall systems. 
However, we do not discuss the thin film limit in this work, 
rather leave it as a subject for future investigation. 

As mentioned earlier, the presence of helical states on the 
side surfaces can, in principle, ruin the perfect quantization 
of Hall conductivity. As an alternative, experiments in a 
Corhino-shaped geometry can be carried out to access vari¬ 
ous Hall states. As shown in Fig. 4, when a time dependent 
fiux 0(0 pierces through the hollow part of the cylindrical 
shaped system, there is a net flow of charge from the inner to 
the outer surface or vice-versa, depending on the direction of 
the applied magnetic field. The net charge accumulation on 
one side of the cylinder A 2(0 is proportional to the Hall con¬ 
ductivity {o-xy) according to A2(0 = AO(0, where AO(0 

is the change of fiux over time t. Such an experiment has been 
carried out successfully in regular 2D electron gas, and the 
signature of various integer as well as fractional Hall states 
has been observed [62]. We believe that similar experiments 
can also be carries out in topological insulators to explore the 
quantum Hall phenomena on its surface. 

The above proposal for quantum Hall measurement in TKIs 
with Corbino geometry can be considered as a generalization 
of Laughlin’s analogous proposal [63] for two dimensional 
electron gas systems, which follows a gauge invariance ar¬ 
gument that the accumulation of charge on one surface (in¬ 
ner or outer depending on the direction of magnetic field) of 
the Corbino disk is proportional to the Hall conductivity in 
the system. The charge accumulation takes place only due to 
the topologically protected chiral edge states, while the heli¬ 
cal states from the side surfaces do not contribute in this pro¬ 
cess [60] . With the recent success of observing quantum Hall 
states on the surface of TIs (B 12803 ) [49], implementation of 
corbino geometry to observe the Hall states may not be nec¬ 
essary to demonstrate similar phenomena in TKIs. However, 
it remains as an interesting proposal to further pin the topo- 



FIG. 4. Corbino-shaped THBs. A time-dependent flux 0(0 pierces 
through the hollow part of the cylinder, and A2(0 is charge accumu¬ 
lated on the outer surface. 


logical nature of the Hall states on the surface of TKIs (in 
particular the fractional Hall states). 


C. Zeeman splitting and uniaxial strain 

Next we explore the effects of various single-particle per¬ 
turbations, such as the Zeeman coupling and external strain 
on the surface states. We argue that Zeeman coupling intro¬ 
duces a mass for these surface Dirac fermions. In addition, 
when a uniaxial strain is applied on the surface along a certain 
direction (F-Z or F-T), the Ising-type symmetry between the 
X and Y valleys can be lifted. 

The Zeeman coupling is given by 

6Hf = rn^cT,, (7) 

where and gx is the effective g-factor for elec- 

Irons at valley A. This perturbation shifts the zeroth Landau 
level to energy m^, and places all other Landau levels (|A| > 1) 

at energies ±E]^{A) to + ^E^(A) m^. Such Dirac mass can 

be further enhanced due to the residual electronic interactions 
on the surface. Hence, even if the bare g-factors in THBs may 
be small, it can be substantially renormalized and enhanced 
by electron-electron interaction, making the Dirac mass rather 
large. 

In order to preserve the symmetry between X and Y val¬ 
leys, we set gx = 8? = §1' However, a different g-factor for 
the f valley carriers, gf = go, is permissible [64], which can 
be justified in the following way. The effective g-factor for the 
surface states is set by the overlap between the wave functions 
of two hybridizing bands with opposite parities and different 
g-factors on the surface [65]. Since, such overlap at X and Y 
points are constrained to be equal due to the underlying cu¬ 
bic symmetry in the bulk, implying g^^ = gy, but generically 
different from that at F point, and thus gp ^ gi, in general. 
The difference between the g-factors can also be enhanced by 
band-bending effects near the surface [42] . 

The Zeeman coupling therefore causes a field-dependent 
energy difference between the zeroth Landau level at the f 
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and XjY valleys, given by 

= EfiN = 0 ) - E^{N = 0 ) = Ao + \g,«EzB, ( 8 ) 

where geff = plays the role of an effective Zeeman 

coupling. 

Notice that neither the orbital nor the Zeeman effect can lift 
the Ising-type symmetry between X and Y valleys. However, 
the application of an external uniaxial strain field on the sur¬ 
face can break this symmetry. The strain field can be applied 
by placing a piezoelectric material on the surface with a volt¬ 
age bias, and reasonable values of Ast can be estimated to be 
about few meV [ 66 ]. For concreteness, we assume that strain 
is applied along the F-F line, so that the surface gets elongated 
(contracted) along the F-F (T-X) direction. Such perturbation 
lowers (raises) the energy of carriers in the X (F) valley [ 66 ], 
and consequently the degeneracy between the X and F valleys 
is lifted, which can be accounted for by setting 

SHf = -Ast, SHf = Ast, (9) 

in Eq. (1). Note that in principle the energy shift between 
X and F valleys can be different in magnitude. Neverthe¬ 
less, due to the cubic symmetry of the system, which trans¬ 
lates into a four-fold rotational symmetry on the (001) sur¬ 
face, the compressibility tensor is expected to by symmetric 
under v ^ y, and consequently However, be¬ 

cause our focus is on the ground state of the system, such a 
simplified assumption is sufficient. We will comment on this 
issue again in Sec. IV C. Aside from developing an energy 
offset between X and F valleys, the uniaxial strain can also 
give rise to /c-dependent perturbations to the effective Hamil¬ 
tonian in Eq. (1), which to lowest order in k, only modifies 
the anisotropy of the three surface Dirac cones according to 

^ ~ Sy = S Sc. (10) 

Such a change in anisotropy does not shift the energy of the 
zeroth Landau level, but changes the wave-functions and con¬ 
sequently can lead to interesting features in the phase dia¬ 
grams of interacting surface states. 

Notice, in the presence of a uniaxial strain that removes the 
degeneracy between X and F valleys, quantum Hall states can 
be observed for |y| = . 

III. ELECTRON-ELECTRON INTERACTION IN LOWEST 
LANDAU LEVEL 

We now focus on the subspace of three zeroth Landau lev¬ 
els (N = 0) on the (001) surface, and study the effects of 
electron-electron interaction in this subspace. Since the ef¬ 
fect of interactions in the quantum Hall regime is particularly 
prominent at integer Landau level fillings [67], we only con¬ 
sider the scenario when one of the three zeroth Landau levels 
is completely occupied. Moreover, due to the particle-hole 
symmetry of a filled Landau level [ 68 , 69], our discussion 
is equally applicable at 2/3 filling in this subspace. Specif¬ 
ically, the set of broken-symmetry phases at 2/3 filling can be 


(a) 




(b) 

X- 


-Y 


1/3 filling 




Y 

2/3 filling 


FIG. 5. Illustration of particle-hole symmetry in the V = 0 Landau 
level subspace. In (a) and (c) the f valley Landau level has a higher 
energy, while in (b) and (d) the situation is reversed. 


one-to-one mapped onto those at 1/3 filling. For example, as 
shown in Fig. 5 the situations in (a) and (b) are related to (d) 
and (c) by particle-hole symmetry, respectively. Thus we will 
not elaborate on the physics at 2/3 filling. 

The central aspects of electron-electron interaction in the 
quantum Hall regime can be captured within a simplified 
framework, if one is allowed to neglect Landau level mix¬ 
ing. Such an approximation can be justified in many 
two-dimensional electronic systems, such as GaAs quantum 
wells [70, 71] and possibly in graphene [72, 73]. We believe 
that this approximation can also be justified for the surface 
states of THBs, for the following reason. In the quantum Hall 
regime the ratio of Coulomb energy to Landau level separation 
is equal to e^ IkHv [73] ~ 14.6//C, for a given Fermi velocity of 
the surface Dirac fermions v ~ 10^ m/s (extracted from quan¬ 
tum oscillation measurements [50]), where k is the dielectric 
constant. Therefore, the regime of small Landau level mixing 
is accessible for experimentally available magnetic fields. 

In the absence of Landau level mixing, the exact ground 
state at integer filling factors is known to be comprised of a 
set of fully occupied Landau level orbitals that share a com¬ 
mon spinor describing the internal degrees of freedom such as 
spin, layer, or valley [67, 70, 71, 74-81]. Some well known 
systems that can support such broken-symmetry states, often 
referred to as quantum Hallferromagnets, are GaAs and AlAs 
quantum wells [ 66 , 82], single and multilayer graphene [83- 
87], surfaces of silicon [ 88 , 89] and SnTe [55], and bulk bis¬ 
muth [90, 91]. Here we demonstrate that in the quantum Hall 
regime, the ground state of surface Dirac fermions in THBs 
is also quantum Hall ferromagnets with various spontaneous 
valley orderings, which, while sharing some similarity with 
other 2D systems, has its own peculiar and interesting fea¬ 
tures. 

In what follows, we will completely neglect the Landau 
level mixing due to any ordering with the zeroth LL. Note that 
LL mixing plays an important role in multi-component sys¬ 
tems such as few-layer graphene, where due to the valley and 
spin degeneracy of the zeroth LL, a degenerate set of broken- 
symmetry phases can develop within this manifold. Thus, one 
needs to account for LL mixing to unambiguously pin the na¬ 
ture of the ground states. In fact, such a mechanism turns 
out to be quite important and decisive within the zeroth LL in 
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few-layer graphene. However, the surface of strong TI only 
hosts non-degenerate zeroth LLs near each valley. Therefore, 
electronic density can only display either valley-polarized or 
valley-coherence distributions for a uniform state. Incorpo¬ 
ration of LL mixing effect can thus only alter the outcomes 
quantitatively, without affecting the qualitative features pre¬ 
dicted in our work. 



Xii 

X 22 

X 12 

^23 

^ = 0 

0.8862 

0.8862 

0.8862 

0.8862 

^ = 0.1 

0.8862 

0.8840 

0.8834 

0.8774 

^ = 0.2 

0.8862 

0.8772 

0.8750 

0.8515 

^ = 0.3 

0.8862 

0.8654 

0.8604 

0.8098 


TABLE I. Typical values for the exchange integral in Eq. (16). 


A. Hartree-Fock Hamiltonian 


Our analysis is based on the Hartree-Fock approximation, 
which happens to be exact at integer Landau level filling fac¬ 
tors for large enough applied magnetic fields [67, 71]. We 
assume that the system has a uniform ground state, so that the 
Hartree contribution cancels with the positive background. As 
a result, the only contribution comes from the Fock (i.e. the 
exchange) term, yielding the mean-field Hamiltonian [71], 

^d) 

qAi 


where S is the surface area of (001) plane, N(p = S jlnfi is the 
orbital degeneracy of each Landau level, and p^j^^are compo¬ 
nents of the density matrix. F are iht form factors [67], 

which, when projected onto the subspace of three = 0 Lan¬ 
dau levels, is given by 


where 




(q) = 


f fi /2 


9 9 

'a, a, 
Aj Ak 


^AjAk 




( 12 ) 


(13) 


In addition, anisotropy of the Dirac cones gives rise to a new 
contribution in 


^AjA, - 


ai a‘ 

Ak 


Ak 


(14) 


In the presence of electron-electron interaction, the ground 
states within this subspace can then be obtained from the 
mean-field Hamiltonian in Eq. (11). In what follows we ob¬ 
tain explicit forms of Hmf for the following two cases: (i) in¬ 
travalley interaction involving small momentum transfer, and 
(ii) intervalley interactions for which the (large) momentum 
transfer is proportional to the separation between the valleys 
in the momentum space. 


1. Intravalley scattering 

Among various interaction channels, the leading contribu¬ 
tion comes from the intravalley interaction, for which scat¬ 


tering between two electrons occurs within the same val¬ 
ley [73]. Contribution from such a process is captured by set¬ 
ting Ti = /I 4 = T and /I 2 = /I 3 = cr in Eq. (11). The mean-field 
Hamiltonian then takes the form 

Kv = -Net -pAAPkrA)XAcr, (15) 

where the superscript “intra” stands for intravalley contribu¬ 
tions. The elements of exchange integral matrix are 

a'F X—^ ^Tte^ 

= - FAA(q)F^A-q), ( 16 ) 

e^S ^ KQ 
q 

after replacing V{q) in Eq. (11) by the 2D Coulomb potential 
(y(q) ~ l\q\). Although actual values of exchange inte¬ 

grals depend on the anisotropy factor 6 of the Fermi surfaces 
(see Table I), some general features of X^cr can be appreci¬ 
ated based on symmetries: (i) The four-fold rotational sym¬ 
metry between X and Y valleys guarantees that A 22 = A 33 and 
X \2 = A 13 . Therefore, X^cr has ovAy four independent ele¬ 
ments, namely An, A 22 , A 12 and A 23 . (ii) The Fermi surface 
ellipticity reduces the strength of the exchange integral [55] 
for long-range Coulomb interaction. Hence, the two diago¬ 
nal elements satisfy A 22 < An in general, and the difference 
An - A 22 increases with increasing d, as can be seen from 
Table 1. 


2. Intervalley interaction or backscattering 

In THBs, orbitals with relatively high angular momen¬ 
tum (such as p, J, / orbitals) participate in the bulk band- 
inversion. Consequently the surface states are also composed 
of orbitals in which Hubbard-type short-range interactions can 
be substantially strong. The presence of such strong resid¬ 
ual short-range interactions on the surface states calls for the 
inclusion of large-momentum backscattering in the theory, 
which enters the Hamiltonian in the form of intervalley in¬ 
teractions [73]. This contribution can be analyzed by taking 
Ai = A 3 = A and A 2 = A 4 = cr in Eq. (11), for which the 
mean-field Hamiltonian takes a rather compact form 

<F = ad 

Ai^cr 

where the superscript “BS” denotes contributions from 
backscattering. The matrix elements Y^cr are given by 

i'i. = (18) 
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where U^cr represents the strength of intervalley scattering 
processes between two valleys A and cr. Although it is chal¬ 
lenging to get an accurate estimation of it may not 
be too unrealistic to take U^o- ~ where K^o- is 

the momentum-space separation between valleys A and cr. 
The underlying rotational symmetry between X and Y valleys 
guarantees that similar to the situation for 

Consequently, the Hamiltonian in Eq. (18) is defined in terms 
of only two independent parameters, Y^ and Y 23 . We also 
note while Y^cr scales as X^cr is independent of the ap¬ 
plied magnetic field. This difference will play an important 
role in determining the ground states of the system. 


+ (8X12 + 8F12 - 4 Xn - 3X22 - X23 - T23)], 
b = % - ^12 - Yu) - %Ao, 

c = 4*:Ao - 4 s/b^EcXu. 

Ec = ^^/25.6nm ^ 56meV is the scale of the Coulomb in¬ 
teraction. The ground state of the system is then obtained by 
minimizing the energy functional in Eq. (21). 


IV. SPONTANEOUS VALLEY ORDERING IN ZEROTH 
LANDAU LEVEL 


B. Mean-field energy functional 


We can now proceed with Eqs. (15) and (17) and arrive at 
the mean-field energy functional within the subspace of three 
zeroth Landau levels. Eor convenience, we introduce a three- 
component spinor 'F = [zi,Z 2 ,Z 3 ] to describe possible ground 
states, where complex variables za = tac^^^ for T = 1,2,3, 
respectively describe the electron occupation in the f, A, and 
Y valleys. The mean-field energy functional then becomes 


, 2 r 3 


£MF = Ar,(Aor^yy]4 


j,k=l 




(19) 


which includes contributions from both the single-particle and 
the Lock terms. The notation Yj' indicates that terms with 
j = k are excluded from the summation. Note that phases 
(6a) of all complex variables za drop out from the mean-field 
energy functional E’mf, indicating that the electron number in 
each valley is separately conserved, and thus the mean-field 
Hamiltonian exhibits aU(l)( 8 )U(l)(S)U(l) symmetry. 

To capture various possible phases in this system, we intro¬ 
duce a new set of variables 


ri = Vl - r 2 = R cos 0, = R sin 0, (20) 


withO < R < 1 and 0 < (f) < 7 il 2 . With such a parametrization, 
R = 0 represents a state polarized in the T valley. On the other 
hand, polarization of electronic density in the X and the Y val¬ 
ley correspond to 7? = 1, with 0 = 0 and 7 r/ 2 , respectively. 
A valley-coherent distribution of electronic density between 
X and Y is represented by R = 1 with 0 < 0 < 7 r/ 2 . Ei- 
nally, 0 < 7? < 1 indicates a coherent superposition between T 
and X or Y valleys. Therefore, different phases can be identi¬ 
fied quite effectively through this parametrization. In terms of 
these new parameters (R and 0), the mean-field energy func¬ 
tional in Eq. (19) assumes a compact form 

Emv{R, 4>)= A{al^+ bR^+ c), (21) 


where 


a = ^[B]_Ec\(X 22 , + T 23 - X 22 ) cos(40) 


We proceed with the mean-field energy functional from 
Eq. (21) and search for possible ground states resulting from 
electron-electron interaction. We also explore the influence of 
Zeeman coupling and strain field. However, it is worth incor¬ 
porating these ingredients one by one, as they influence the 
phase diagram differently. 


A. Pristine surface without Zeeman or strain field 

Let us begin the discussion by turning off Zeeman cou¬ 
pling and external strain field, and solely focus on the effects 
of electron-electron interaction. This exercise will provide 
valuable insights into the overall structure of the phase dia¬ 
gram of interacting surface states, as shown in Eig. 6 . We 
first keep only the contributions from intravalley interaction, 
which amounts to setting all Yact to zero in Eq. (21). In such a 
limit, electrons maximize the exchange energy gain by keep¬ 
ing only one of three valleys fully occupied. Therefore, the 
ground state must always be valley polarized. In particular, 
in the absence of an energy offset among the valleys (i.e., 
Ao = 0 ), electrons must occupy the f valley, which continues 
as the preferred ground state for arbitrary Aq < 0, as shown by 
the shaded region in Eig. 6 (a) for d = 0.2. Such an outcome 
can be anticipated intuitively, since a circular Eermi surface 
acquires maximal exchange energy from the long range tail of 
the Coulomb interaction (notice Xn > X 12 in Table I for any 

It is useful to contrast the current situation with 2D sys¬ 
tems like graphene [67] and monolayer M 0 S 2 [54], where 
all Eermi pockets are isotropic. In those systems intravalley 
Coulomb interaction by itself cannot determine the ground 
state uniquely. This outcome can be appreciated from the 
energy functional Eq. (19) if we set both 6 and Yact to zero. 
Under that circumstance, all entries from the intravalley ex¬ 
change integral are equal [see Table I], and the contributions to 
the energy functional Eq. (19) become constant, independent 
of the ground state configuration. Indeed, the actual ground 
state in graphene is likely to be determined by interaction ef¬ 
fects at the lattice scale [72, 87, 92-95]. 

When the noninteracting zeroth Landau level near f valley 
is placed at a higher energy than those residing near X and 
Y valleys (i.e., Aq > 0), the single-particle energy offset (Aq) 
competes with the exchange energy, and there exists a first- 
order phase transition from a X (or F)-valley polarized state 
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B(T) 



FIG. 6. Phase diagram within the zeroth Landau level of interacting 
surface states, residing on the (001) surface of THBs in the presence 
of (a) intravalley scattering and (b) both intravalley and backscatter- 
ing interactions. We here set 6 - k - 2, and the backscattering 
interaction Uxo- - 2.ne^ j Red (black) lines represent first-order 
(continuous) phase transition. The red dot in (b) marks the criti¬ 
cal field defined in Eq. (23). The dashed lines are obtained for 
6 = 0.25, while keeping the rest of the parameters in the theory un¬ 
altered. 


to a f-polarized state as the field strength is increased. No¬ 
tice that X and Y valleys are always degenerate and therefore 
the X- (or ?-) valley polarized state spontaneously breaks an 
Ising symmetry. The finite-temperature phase transition into 
this phase belongs to the two-dimensional Ising universality 
class, and the ground state represents a quantum Hall Ising 
nematic order, as the four-fold rotation symmetry on the sur¬ 
face is spontaneously broken. 

The boundary of the aforementioned first-order transition is 
determined by 

^o = E,{Xn-X22}^^K|K, ( 22 ) 

in B-Xq plane and shown by the red lines in Fig. 6(a) for 
6 = 0.2 (solid line) and 6 = 0.25 (dashed lineb). The square- 
root field dependence of this phase boundary stems from the 
^^B scaling of the exchange energy from intravalley interac¬ 
tions. With increasing ellipticity of X and Y valleys (d), the 
difference between diagonal components of the exchange inte¬ 
grals (Xii -X 22 ) increases and the phase boundary gets shifted 
upward in the B-Aq plane [compare the solid and dashed lines 
in Fig. 6(a)]. 


With the inclusion of backscattering interaction, the possi¬ 
bility of realizing a valley-coherent distribution of electronic 
density also arises. In the presence of backscattering, the line 
of first-order transition ceases to exist beyond a critical field 
[shown by the red dot ^ in Fig. 6(b)], and bifurcates into 
two lines of continuous phase transitions [shown by the black 
lines in Fig. 6(b)]. The critical field is given by 

liTT^e^ O 

4 ^ (Xn + X22 - 2 Xnf, (23) 

^12 

yielding ~ 1.7T and 4.3 T, respectively for 6 = 0.2 and 
0.25, and U \2 = Ine^IK\ 2 - The quantity in the parentheses in 
Eq. (23) increases with 6 , and hence the critical field B^P also 
increases as the pockets near X and Y points become more 
elliptic. 

The lines of continuous phase transitions in the 5-Ao plane 
are obtained from 

2 'sfBEr ( 6^12 \ 

(24) 

2 H\2 \ 

Equation (24) defines the phase boundary between a pure f- 
polarized state and a state with coherent superpositions be¬ 
tween f and X/F-valley. In contrast, the phase boundary be¬ 
tween Y-X/Y coherent phase and pure XjY polarized state is 
determined from Eq. (25). Notice that unlike the transitions 
between valley-polarized phases, these phase transitions are 
deemed continuous, since the ground-state wave function does 
not display any abrupt changes across these boundaries. 

Eor the valley-coherent phase, shown in Eig. 6, aside from 
the Z 2 -Ising symmetry between X and Y valleys, the con¬ 
tinuous U(l) symmetry associated with the conservation of 
electronic density at each valley is also spontaneously broken. 
As a result, the collective excitations in such valley-coherent 
phase are gapless Goldstone modes. 

In principle, it is conceivable to realize a ground state where 
the electronic density is coherently shared between X and Y 
valleys. The critical field (5p^) above which such a phase 
may develop in the system is 

=^^{X 22 -X 22 )\ (26) 

^23 

However, with a reasonable estimation of intervalley interac- 
tion U 23 ~ 2ne^ IK 23 , we find 5® = 50T for (5 = 0.2. This 
is the reason such a phase does not appear in Eig. 6(b). Rel¬ 
atively high value of (in comparison to B^P) originates 
from two sources. Eirst, the separation between X and Y val¬ 
leys is larger than that between f and X ox Y valleys. There¬ 
fore, it is natural to anticipate that the intervalley interaction 


^ Note that the point marked by this red dot is actually a tricritical point in 
the phase diagram, at which a line of first-order transition terminates and 
two lines of continuous phase transition emanate. 
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FIG. 7. Phase diagram with (a) only intravalley and (b) both in¬ 
travalley and backscattering interactions for various values of geff- 
For ^eff = 0. we recover the phase diagram without the Zeeman ef¬ 
fects [the red line Fig. 6(a)]. In (b) we set the geff = 2, and the dashed 
lines reproduce the phase boundaries of Fig. 6(b). We set S = 0.2 and 
/c = 2 for all plots. 


between X and Y pockets, U 23 , is weaker than Uu. Second, 
the difference among the exchange integrals appearing on the 
right-hand-side of Eq. (26) is also much larger than that of 
Eq. (23). 


B. Phase diagram with Zeeman coupling 

We will now discuss the effects of Zeeman coupling on the 
phase diagram. We show that it does not alter the qualitative 
nature of the phase diagram, instead plays the role of a tuning 
parameter that allows us to access various regions of the phase 
diagram. Our findings are summarized in Eig. 7, which are 
obtained by replacing the energy offset Aq in Eq. (19) by 
(energy offset after incorporating the Zeeman coupling) from 
Eq. (8). We note that, although the Zeeman coupling is weak 
at small fields, it becomes important as the magnetic field gets 
stronger since it grows faster with the applied field than the 
intravalley Coulomb energy. 

Indeed, the results in Eig. 7 fulfill this expectation. When 
the effective Zeeman coupling geff = gf - gxi? > it can 
cause a transition out of the E-polarized state into an X- ovY- 
polarized state for a moderately strong magnetic field, which 
was previously inaccessible just by tuning the field strength 
[see Eig. 6(a)]. Such an effect is transparent in Eig. 7(a), where 


the intervalley interaction is switched off. Even when we in¬ 
corporate backscattering, the qualitative structure of the phase 
diagram remains unchanged, as shown in Eig. 7(b). How¬ 
ever, the regime of X- or F-valley polarized as well as E- 
XjY valley coherent phases increases in the phase diagram 
with increasing geff- Thus, by changing the Zeeman coupling, 
which one can tune to a certain degree by tilting the magnetic 
field away from the perpendicular orientation (since the Zee- 
man coupling depends on the total magnetic field) on (001) 
surface, various regions of the phase diagram and associated 
phase transitions can be accessed. In contrast, the situation for 
gefF < 0 is less interesting, as the phase diagram does not qual¬ 
itatively differ from that obtained without the Zeeman cou¬ 
pling. 


C. Effects of uniaxial strain field 


Einally, we investigate the effects of strain on the phase di¬ 
agram. We restrict ourselves with a uniaxial strain applied 
along the F-F direction, which breaks the rotational symme¬ 
try between X and F valleys. Hence, the application of such 
a uniaxial strain lifts the Ising-type symmetry between X and 
F valleys. As shown in Sec. IIC, such a strain field has two 
distinct consequences. We argue that both of them favor X 
valley polarization as the ground state of the system. Note 
that due to the underlying cubic symmetry of the system, we 
have assumed the strain-induced energy shifts to be equal in 
magnitude for X and F pockets, as shown in Eq. (9). Even 
for the sake of argument, we allow the shifts in energy to be 
different for X and F valleys, the valley, pushed to the higher 
energy, will always be excluded from the ground state mani¬ 
fold, irrespective of its magnitude. 

The first one arises from a single-particle effect that in¬ 
creases (decreases) the energy of the F (X) valley by Ast 
[see Eq. (9)] and thus drives the F valley out of the ground 
state manifold. The second one is more subtle. As shown 
in Eq. (10), the applied strain field reduces (increases) the 
anisotropy of the Dirac cones at X (Y) valley. According to 
Table I, this effect amounts to increasing (reducing) the ex¬ 
change energy gain for electrons in the X (Y) valley, which 
further pushes the F valley to a higher energy (relative to the 
X valley). Consequently, the F valley no longer appears in the 
ground state phase diagram when a uniaxial strain is applied 
along F-F on the surface, as shown in Fig. 8. In addition, 
in comparison with Fig. 7 we find that the strain field gives 
rise to a larger phase space for the A-valley polarized and f -X 
coherent phases. Such an outcome stems from the fact that 
uniaxial strain (applied along F-X or F-Y) also turns the F val¬ 
ley into an elliptic one, thus reducing its energy gain from 
electron-electron interaction [see Eq. (10)] (also note that the 
ellipticity of X valley is reduced at the same time). Therefore, 
an applied strain on the surface of THBs serves as a useful 
tuning parameter to explore various broken symmetry phases. 
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FIG. 8. Phase diagrams with Zeeman coupling and uniaxial strain 
fields for (a) pure intravalley and (b) both intravalley and backscat- 
tering interactions. Perturbations from strain fields are Ast = 0.1 meV 
and Sc = 0.02, which places the Y valley at a higher energy. Rest of 
the parameters are the same as those in Fig. 7. The dashed lines in 
(b) reproduces the solid lines in Fig. 6(b) for comparison. 


D. Pure short-range interactions 

Before concluding this section a discussion on screening of 
long-range Coulomb interaction seems appropriate. Let us re¬ 
place the long-range Coulomb potential in Eq. (15) by a short- 
ranged one Iko, where the constant ko characterizes the 

strength of the short-range interaction. With such an interac¬ 
tion potential, all diagonal elements of the exchange integral 
are identical, with Xn = X 22 = ^33 = Ijilkoi). The 
off-diagonal elements are still smaller than the diagonal ones, 
given by Xi 2 =Xi 3 = V6/(5;toO and Z 23 =6/(13;toO- There- 
fore, when the long-range Coulomb interaction is replaced by 
a short-ranged one, circular Fermi pockets lose the energetic 
advantage over elliptic ones. Consequently, when the energy 
off-set Ao is zero, the three valleys always remain degenerate, 
and the electronic density spontaneously chooses one of them 
to condense into at low temperatures. The finite-temperature 
phase transition into such a broken-symmetry ground state is 
described by the three-state Potts model. Near the phase tran¬ 
sition the correlation length exponent is v = 5/6 and the order 
parameter anomalous dimension is 7/ = 4/15 [96]. In con¬ 
trast, for arbitrary Aq < 0 and Aq > 0 the ground state is 
respectively f-valley and X/F-valley polarized. Thus, with 
pure short range interaction and for Aq > 0, the ground state 


corresponds to a quantum Hall Ising-nematic order. 


V. DISCUSSION AND CONCLUSION 

To summarize, in this work we have studied the interplay 
of external magnetic fields (as well as strain) and electron- 
electron interaction on the two-dimensional surface states of 
a family of heavy fermion hexaboride compounds which are 
predicted to be topological insulators [97]. Systems that may 
support such topologically nontrivial ground states at low tem¬ 
peratures are SmB6, YbB6, PuB6. These materials have a bulk 
insulating gap, share a common cubic symmetry, and the band 
inversion (requisite criteria for an insulator to be topological) 
occurs around three high-symmetry X points in the bulk Bril- 
louin zone. Consequently, the surface states are composed 
of three copies of massless Dirac fermions, referred to as the 
valleys. For concreteness, we here focused only on the high 
symmetry (001) surface of THBs. 

Subject to strong magnetic fields, these surface states sup¬ 
port three copies of Dirac Landau levels. We show that de¬ 
pending on the relative energies of the Dirac points and their 
effective Fermi velocities, a variety of integer quantum Hall 
states can be realized (including contributions from both top 
and the bottom surfaces that simultaneously undergo Landau 
level quantization). In particular, due to the two-fold degener¬ 
acy between X and Y valleys, we expect an absence of quan¬ 
tum Hall states at some filling factors. However, the appli¬ 
cation of a uniaxial strain (along T-X or L-F) removes such 
a degeneracy, and quantum Hall states at all (integer) filling 
factors should be accessible. Robust quantum Hall states can 
be observed from the quantization of Hall conductivity [49] or 
the time-dependence of charge transfer from the inner to the 
outer edge in a Corbino-shaped system [62]. 

The electron-electron interaction is shown to support a rich 
variety of broken symmetry phases within the zeroth Lan¬ 
dau level. For example, if we consider only the long ranged 
tail of the Coulomb interaction [y(q) ~ ^^/|q|], the ground 
state is always a valley polarized quantum Hall ferromagnet. 
Long-range Coulomb interaction prefers to keep the f val¬ 
ley fully occupied due to the circular Landau orbitals in its 
vicinity. Due to a four-fold rotational symmetry, X- and Y- 
valley polarized states are energetically degenerate. Thus, at 
low temperatures when X/Y valley is filled, electronic density 
spontaneously breaks a valley-Ising symmetry and the ground 
state should represents a quantum Hall Ising nematic order. 
In contrast the short-range intervalley interactions can give 
rise to valley-coherent states, where electronic density is dis¬ 
tributed among two valleys. While the transition between the 
two valley-polarized states is always first order in nature, that 
between a valley-coherent and valley-polarized state is con¬ 
tinuous. 

We also address the effects of Zeeman coupling and appli¬ 
cation of a uniaxial strain on the phase diagram of interacting 
surface states. Neither of these two external fields changes 
the qualitative structure of the phase diagram but modifies the 
relative phase space available for distinct symmetry breaking 
phases (valley coherent and polarized). Therefore, both Zee- 
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man coupling (which can be tuned efficiently by tilting the 
magnetic field) and the uniaxial strain (introduced by placing 
a piezoelectric material on the surface) serve as useful tuning 
parameters to access various ordered phases as well as phase 
transitions in experiments. However, uniaxial strain breaks 
the Ising-type symmetry among X and Y valleys and acts as 
an external field to the order parameter (electronic density in 
one valley). Therefore, a uniaxial strain prefers one of these 
two valleys as the ground state. 

Detection of the precise ground state in multi-valley sys¬ 
tems can be challenging. We believe that a measurement of 
the longitudinal resistivity (Rjj), where j = x ov y {z is the 
direction perpendicular to the surface), at finite temperatures 
and its scaling with the magnetic field can provide distinguish¬ 
ing signatures for various broken-symmetry phases. For ex¬ 
ample, Rjj is expected to be isotropic in the f-polarized phase, 
and thus XR = IRx^-Ryyl = 0. In contrast, an X or F-polarized 
state should display an anisotropy between Rxx and Ryy, but 
AR in this phase is independent of the magnetic field strength. 
The valley-coherent phase (either T-X or f-F) also gives rise 
to anisotropic longitudinal resistivity. However, the relative 
occupation of electronic density between the two valleys (T 
and X or ?) changes with the magnetic field. Therefore, AR 
is expected to change with the strength of the magnetic field 
in a valley-coherent phase. Hence, from the scaling of AR 
with magnetic field, it should be possible to figure out the ex¬ 
act pattern of symmetry breaking in the ground state. Similar 
proposals have been discussed to detect nematic valley order¬ 
ing in other quantum Hall systems [98, 99]. 

We now discuss an interesting internal structure of the 
zeroth Landau level that is solely tied with the topology 
of the surface Dirac cones. While arriving at Eq. (1), we 
have assumed that all three Dirac cones have identical spin- 
momentum locking. Consequently, three copies of zeroth 
Landau levels have identical spin polarizations. However, 
it is conceivable that spin-momentum locking of the Dirac 
cones near the f valley is opposite to that near X and Y val¬ 
leys. This scenario is protected by the underlying nontriv¬ 
ial Z 2 topological invariant in the bulk and can be accom¬ 
plished by tuning the relative strength of nearest-neighbor 
and next-nearest-neighbor hybridization among the opposite 
parity bands [42, 100]. When spin-momentum locking at T 
and X/Y points is opposite, the spin-polarization of the ze¬ 
roth Landau levels near these valleys is also opposite. Con¬ 
sequently, the T-polarized and Z/F-polarized states are char¬ 
acterized by opposite spin-polarization. An even more inter¬ 
esting situation arises in the valley-coherent phase, where the 
electronic density is shared between two zeroth Landau lev¬ 
els from different valleys. Since the zeroth Landau level at 
the two valleys now carries opposite spin angular momentum, 
the valley-coherent state corresponds to sl ferrimagnet, with 
finite uniform and staggered magnetization. In this situation, 
effects of uniaxial strain remain unchanged, but significant 
modifications can arise from the Zeeman coupling. Since the 
g-factors within the zeroth Landau level at T-valley (go) and 
X/F-valley (gi) are of opposite signs, the system can experi¬ 
ence a large effective g-factor (geff = go - gi) even for small 
go and gi. Therefore, various regions of the phase diagrams 


in Figs. 7 and 8 can now be accessible even for weaker mag¬ 
netic fields. Even when the zeroth Landau levels near different 
valleys carry opposite spin polarization, measurements of AR 
can distinguish various valley polarized and valley-coherent 
states. However, spin-polarized STM and magnetic exchange 
force microscopy experiments can precisely obtain the spin 
structures of the ground states [101]. 

Although we here only accounted for the long-range 
Coulomb and the short-range backscattering interactions, as 
they are typically the most important contributions in a pris¬ 
tine system, our analysis can be generalized to explore some 
additional possibilities. For example, lattice-scale effects and 
electron-phonon interactions can support more exotic ground 
states, such as nematic (Pomeranchuk-type), stripe, transla¬ 
tional symmetry breaking density-wave orders. In addition, 
our theory can also be extended to study a topological in¬ 
sulator thin film, which resembles the conventional double¬ 
layer quantum Hall system in GaAs [75], where a plethora 
of broken symmetry phases associated with the layer degrees 
of freedom, such as a canted antiferromagnetic state [76-78], 
can be realized. We leave this topic for THBs for a future 
investigation. 

Even though in this work we completely focused on uni¬ 
form ground states, the excitations above the ground state 
can carry a nontrivial topological structure, such as the valley 
skyrmions. It is generally expected [67] that at integer Landau 
level fillings, skyrmions can compete with simple particle- 
hole pairs for the lowest-energy charged excitations. If the 
system is placed slightly away from the integer filling factors, 
the skyrmions may also constitute the ground-state configu¬ 
ration in this system. Therefore, we expect that in the vicin¬ 
ity of (the tricritical points in Fig. 6), where two valley- 
polarized phases and a valley-coherent phases meet, valley 
skyrmions can become energetically favorable. However, due 
to the anisotropy of the cyclotron orbits the energetic analy¬ 
sis of skyrmion excitations is an involved task, and deserves a 
separate quantitative investigation. 

Finally we comment on the role of disorder. Notice that 
the f-polarized state does not break any discrete symmetry on 
the surface and hence is expected to be robust against disor¬ 
der. In contrast, the X- or F-polarized, and f-A or F-F co¬ 
herent states break the four-fold rotational symmetry, hence 
any generic point-like impurity will couple as random field 
with electronic density when either A or F valley is occu¬ 
pied (fully or partially) [102]. Such random field coupling 
will lead to a proliferation of domain walls of opposite valley 
polarization/coherence at finite temperatures and destroy true 
long range ordering. However, the quantum Hall experiments 
are typically carried out at few hundred millikelvins, where 
the formation of domain walls can be energetically costly. 
In addition, the application of uniform strain can reduce the 
strength of such random field coupling, since it lifts the valley- 
Ising symmetry explicitly, providing stability to A-polarized 
and T-X coherent phases against point-like impurities. It is 
worth pointing out that when F and A, F valleys carry oppo¬ 
site spin-momentum locking, the valley coherent phase carries 
a net magnetic moment, and time-reversal invariance preserv¬ 
ing point-like impurities cannot couple with the ferrimagnet 
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order parameter as random field. Thus, we expect that on a 
sufficiently clean surface of THBs, where residual electron- 
electron interaction is significantly strong, various phases we 
propose in this work can be observed in the quantum Hall 
regime, at least at low temperatures. 
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